The description of the related art of the present invention begins with the description of "Augmented Reality", followed by the description of related known arts.
"Augmented Reality" (hereinafter called "AR"), a concept introduced in 1993, is described, for example, in Document (1), "Communications of the ACM" Vol. 36, No. 7. PP. 24-97.
In contrast to "Virtual Reality" in which all the real world objects are realized virtually, AR augments the real world with computers to realize an easy-to-understand interface.
Among the systems designed based on AR, "Digital Desk" and "InteractiveDESK" are related to the present invention. They are described below.
"Digital Desk" is an interactive desktop system proposed by Pierre Wellner of Rank Xerox corporation. The concept is detailed, for example, in Document (2) "Communications of the ACM" Vol. 36, No. 7, pp. 87-96. As shown in FIG. 8, this system has a projector 202 and a camera 201 above a work desk 203. The projector 202 projects images, generated by a computer 200, onto the work desk 203 (see a projected image 204), and the camera 201 takes pictures of hand motions, hand-written characters, hand-written pictures, and documents. The user can perform tasks interactively while conversing with the computer 200 through the work desk 203 on which the images generated by the computer 200 and real documents are mixed. The so-called real world interface is thus realized. Applications are available which illuminate, and display marks on, the numbers in a document pointed by a finger and paste those numbers on a desktop calculator created by the projected light.
Patent documents related to "Digital Desk" of Rank Xerox corporation include Japanese Patent Publication Kokai JP-A No. Hei 7-168949 (Title of invention "Interactive Copier") and Japanese Patent Publication Kokai JP-A No. Hei 9-237349 (Title of invention "Interactive Desktop System").
Of these patent publications, Japanese Patent Publication Kokai JP-A No. Hei 7-168949 (Title of invention "Interactive Copier") discloses a configuration, comprising a a video projector displaying virtual world information onto a work desk and a camera monitoring the work desk. The disclosed configuration is designed to allow the user to interact with information projected by the video projector and to create documents or pictures on actual paper in the manual operation mode. Thus it enables an interaction between virtual world images, projected by the video projector, and real world paper on the work desk to be performed intuitively in the manual operation mode.
Japanese Patent Publication Kokai JP-A No. Hei 9-237349 (Title of invention "Interactive Desktop System") also discloses a configuration comprising a video projector displaying virtual world information onto a work desk and a camera monitoring the work desk. This configuration allows the user to request the camera to take the picture of real objects, to display via the computer images taken by the camera onto the video projector, and to copy and paste real objects such as documents while sitting at the work desk. In this configuration, a mouse is used to specify an area to be photographed, and the cursor which is a pointing position indicator is projected on the work desk as the display information from the video projector.
The concept and the features of "InteractiveDESK", a system which uses the entire desk as the interactive environment, are detailed, for example, in Document (3), ITE Technical Report Vol. 19, No. 65, pp. 1-6.
FIG. 9 is a decigram showing an overall configuration of InteractiveDESK. As shown in FIG. 9, the InteractiveDESK comprises a work desk 304 whose working surface is a large display through which data may be entered with a pen, an upright display 302 provided in front of the user, a camera 301 monitoring the top of the work desk, and other components such as a keyboard 306 and a mouse 307. The camera 301 recognizes user actions or the positions of documents on the desk and helps the user with his or her work. For example, it switches input means (keyboard 306, mouse 307, or pen) between two types of displays, desk built-in type and the upright type, and extracts electronic information on the paper documents on the desk.
Japanese Patent Publication Kokai JP-A No. Hei 6-266498 (Title of Invention: Data Processing System) is available for a patent application publication relating "InteractiveDESK". Japanese Patent Publication Kokai JP-A No. Hei 6-266498 mentioned above discloses a method for determining a position at which desired information is to be displayed based on at least one of the three: user status, display surface status, and display status.
For example, when documents or some other objects are placed on the display screen, this system displays information at a position where there is no such objects to prevent real world objects such as documents from interfering with the virtual world such as display information. The publication also discloses a method for switching the display with the above-described input means.
Other prior arts similar to "Digital Desk" or "InteractiveDESK" include Japanese Patent Publication Kokai JP-A No. Hei 9-319556 (Title of Invention: Data Processing Device). This publication discloses a system comprising a work desk, a projector projecting images on the work desk, and a camera monitoring the desk and a technology for interacting with real world objects such as documents on the desk. However, there is no distinct difference between the system disclosed in this publication and those systems described above such as "Digital Desk" or "InteractiveDESK". Thus, explanation on the system of Japanese Patent Publication Kokai JP-A No. Hei 9-319556 is obtained herein.
The technologies for pointing to real world objects with laser beam, such as those disclosed in Japanese Patent Publication Kokai JP-A No. Hei 9-46776 (Title of Invention: Remote Indication Device and Method) and Japanese Patent Publication Kokai JP-A No. Hei 5-83705 (Title of Invention: Still Picture Input Device) are outlined below.
FIG. 10 shows the configuration of the remote indication device proposed by Japanese Patent Publication Kokai JP-A No. Hei 9-46776. An indication device 400 and a remote unit 406 are connected via a network 405. The remote unit 406 takes a picture of a document 407 at a remote location and sends the image to the indication unit 400 over the network 405. The indication device 400 displays the image of the document sent from the remote unit 406 onto a display 401. The presenter points to a desired position on the remotely-located document on the display 401 with a keyboard 402 or a mouse 404 connected to the indication device 400. The pointing information is sent to the remote unit 406 via the network 405. In response to the pointing information sent from the indication device 400, the remote unit 406 projects laser beam onto the actual position on the document 407 corresponding to the position pointed by the presenter on the display 401.
FIG. 11 shows the configuration of the still image input device disclosed in Japanese Patent Publication Kokai JP-A No. Hei 5-83705 described above. This device is a pointing apparatus in a pictorial art camera used to present materials to a TV conference. The user operates a pointing device 505 to send pointing information to a CPU 501. In response to the received pointing information, the CPU 501 controls a laser beamemitting device 503 to project laser beam onto a document desk 504. This enables a presenter of a conference to point to actual objects such as conference materials. This technology has an image read instruction area previously allocated on the document desk so that a camera 502 can read an image in the area when the user points it and clicks in it with laser beam.
As the technology for pointing to a desired position with laser beam, laser pointers for presentation are known and many are commercially available. These are used by presenters to indicate images projected by a projector. Pen-sized pointers have been developed.